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Are Disagreements Honest? 


ABSTRACT 

We review literatures on agreeing to disagree and on the rationality of differing priors, in 
order to evaluate the honesty of typical disagreements. A robust result is that honest 
truth-seeking agents with common priors should not knowingly disagree. Typical 
disagreement seems explainable by a combination of random belief influences and by 
priors that tell each person that he reasons better than others. When criticizing others, 
however, people seem to uphold rationality standards that disapprove of such self- 
favoring priors. This suggests that typical disagreements are dishonest. We conclude by 
briefly considering how one might try to become more honest when disagreeing. 
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1. Introduction 


People disagree all of the time, especially about politics, morality, religion, and relative 
abilities. Virtually any two intelligent people can quickly find many topics of 
disagreement. Disagreements usually persist, and often become stronger, when people 
become mutually aware of them. Nor is disagreement usually embarrassing; it is often 
worse to be considered a “fence-sitter” without distinctive opinions. 

Not only do people disagree; they often consider their disagreements to be about what is 
objectively true, rather than about how they each feel or use words. Furthermore, people 
often consider their disagreements to be honest, meaning that the disputants respect each 
other’s relevant abilities, and consider each person’s stated opinion to be his best estimate 
of the truth, given his information and effort.^ 

Yet according to well-known theory, such honest disagreement is impossible. Robert 
Aumann (1976) first developed general results about the irrationality of “agreeing to 
disagree.” He showed that if two or more Bayesians would believe the same thing given 
the same information (i.e., have “common priors”), and if they are mutually aware of 
each other's opinions (i.e., have “common knowledge”), then those individuals cannot 
knowingly disagree. Merely knowing someone else’s opinion provides a powerful 
summary of everything that person knows, powerful enough to eliminate any differences 
of opinion due to differing information. 

Aumann’s impossibility result required many strong assumptions, and so it seemed to 
have little empirical relevance. But further research has found that similar results hold 
when many of Aumann’s assumptions are relaxed to be more empirically relevant. His 
results are robust because they are based on the simple idea that when seeking to estimate 


* In this paper we consider only truth-seeking at the individual level, and do not attempt a 
formal definition, in the hope of avoiding the murky philosophical waters of “justified 
belief.” 
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the truth, you should realize you might be wrong; others may well know things that you 
do not. 

For example, this theory applies to any dispute that can be described in terms of possible 
worlds. This happens when people agree on what the answers would be in each 
imaginable world, but argue over which of these imaginable worlds is the real world. 

This theory can thus apply to disputes about facts that are specific or general, hard or 
easy to verify. It can cover the age of a car, the correctness of quantum mechanics, 
whether God created the universe, and which political candidate is more likely to induce 

prosperity. It can even apply to morality, when people believe there are objectively 

2 

correct answers to moral questions. 

One of Aumann’s assumptions, however, does make a big difference. This is the 
assumption of common priors, i.e., that agents with the same information must have the 
same beliefs. While some people do take the extreme position that that priors must be 
common to be rational, others take the opposite extreme position, that any possible prior 
is rational. In between these extremes are positions that say that while some kinds of 
priors and prior differences are rational, other kinds are not. 

Are typical human disagreements rational? Unfortunately, to answer this question we 
would have to settle this controversial question of which prior differences are rational. 

So in this paper, we consider an easier question: are typical human disagreements honest? 
To consider this question, we do not need to know what sorts of differing priors are 
actually rational, but only what sorts of differences people seem to think are rational. If 
people mostly disagree because they systematically violate the rationality standards that 


^ The economics literature on disagreement is cited in more detail throughout the paper. 
Philosophers have long discussed disagreement, starting with Sextus Empiricus (2000, 
first edition predates 235 A.D.), who argued that when people disagree, they face an 
"equipollence" of reasons, and cannot judge their own perspective to be superior to that 
of others. See also Arnauld and Nicole, (1996 [1683]), Thomas Reid (1997 [1764]), 
Schiller (1934), Brandt (1944), Coady (1992), Nozick (1993), Rescher (1993), Everett 
(2001), and Goodin (2002). 
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they profess, and hold up for others, then we will say that their disagreements are 
dishonest. 

After reviewing some stylized facts of disagreement, the basic theory of disagreement, 
how it has been generalized, and suggestions for the ways in which priors can rationally 
disagree, we will consider this key question of whether, in typically disagreements, 
people meet the standards of rationality that they seem to uphold. We will tentatively 
conclude that typical disagreements are best explained by postulating that people have 
self-favoring priors, even though they disapprove of such priors, and that self-deception 
usually prevents them from seeing this fact. We end by outlining some of the personal 
policy implications that would follow from this conclusion that human disagreement is 
typically dishonest. 

II. The Phenomena of Disagreement 

Before considering the theory of disagreement, let us now review some widely 
recognized stylized facts about human disagreement. Some of these “facts” may well be 
wrong, but it seems highly unlikely that most of them are wrong. 

Virtually any two people capable of communication can quickly find a topic on which 
they substantially disagree. In such disagreements, both sides typically believe 
themselves to be truth-seekers, who honestly say what they believe and try to believe 
what is true. Both sides are typically well aware of their disagreement, and can reliably 
predict the direction of the other’s next statement of opinion, relative to their own last 
statement. 

Disagreements do not typically embarrass us. People are often embarrassed to discover 
that they have visibly violated a canon of rationality like logical consistency. Upon this 
discovery, they often (though not always) change their views to eliminate such violations. 
And in many cases (though again not always) fewer such violations tend to be discovered 
for individuals with higher IQ or superior training. Disagreements, however, happen 
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even though people are usually well aware of them, and high-IQ individuals seem no less 
likely to disagree than others. Not only are disagreements not embarrassing, more social 
shame often falls on those who agree too easily, and so lack “the courage of their 
convictions.” 

Real world disagreements seem especially frequent about relative abilities, such as who is 
smarter than whom, and about subjects, like politics, morality, and religion, where most 
people have strong emotional reactions. Discourse seems least likely to resolve 
disagreements of these kinds, and in fact people often move further away from each 
other’s views, following a sustained dialogue.^ 

Psychologists suggest that human disagreements typically depend heavily on each person 
believing that he or she is better than others at overcoming undesirable influences on their 
beliefs (e.g., innuendo), even though people in fact tend to be more influenced than they 
realize (Wilson, Gilbert, & Wheatley 1998). Many people dismiss the arguments of 
others, often on the grounds that those others are less smart, knowledgeable, or otherwise 
less able. At the same time, however, such people do not typically accede to the opinions 
of those who are demonstrably equally or more able, be the relevant dimension IQ, life 
experience, or whatever. People are usually more eager to speak than they are to listen, 
the opposite of what a simple information-collection model of discussion would predict 
(Miller 2000). 

The positions taken in many disagreements seem predictable, as in the saying that “where 
you stand depends on where you sit.” In general, people seem inclined to believe what 
they "want to believe." For example, most people, especially men, estimate themselves 
to be more able than others and more able than they really are (Waldman 1998). 

Gilovich (1991, p.77) cites a survey of university professors, which found that 94% 
thought they were better at their jobs than their average colleagues. A survey of 
sociologists found that almost half said they expected to become among the top ten 


-1 

On the tendency for polarization, see Sunstein (1999). 
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leaders in the field (Westie 1973)."^ People also tend to think more highly of their 
groups, such as their home team’s sporting success, their nation’s military success, and 
their profession’s social and moral value. 

III. The Basic Theory of Agreeing to Disagree 

Let us now consider the existing theory on disagreement. Most analysis of disagreement, 
like most analysis of inference and decision-making in general, has used Bayesian 
decision theory. Bayesian theory may not be fully satisfactory or fully general, but the 
core results in agreeing to disagree have been generalized beyond Bayesian agents to a 
greatf extent. Such generalizations have been possible because these core results rest 
mainly on a few simple intuitions. To see the intuitive appeal of the basic argument, 
consider the following simple parable. 

Imagine that John hears a noise, looks out his window and sees a car speeding away. 
Mary also hears the same noise, looks out a nearby window, and sees the same car. If 
there was a shooting, or a hit-and-run accident, it might be important to identify the car as 
accurately as possible. 

John and Mary’s immediate impressions about the car will differ, due both to differences 
in what they saw and how they interpreted their sense impressions. John’s first 
impression is that the car was an old tan Ford, and he tells Mary this. Mary’s first 
impression is that the car was a newer brown Chevy, but she updates her beliefs upon 
hearing from John. Upon hearing Mary’s opinion, John also updates his beliefs. They 
then continue back and forth, trading their opinions about the likelihood of various 
possible car features. (Note that they may also, but need not, trade evidence in support of 
those opinions.) 

If Mary sees John as an honest truth-seeker who would believe the same things as Mary 
given the same information (below we consider this "common prior" assumption in 

For a survey of the psychology literature on this point, see Paulhus (1986). 
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detail), then Mary should treat John’s differing opinion as indicating things that he knows 
but she does not. Mary should realize that they are both capable of mistaken first 
impressions. If her goal is to predict the truth, she has no good reason to give her own 
observation greater weight, simply because it was hers. 

Of course, if Mary has 20/20 eyesight, while John is nearsighted, then Mary might 
reasonably give more weight to her own observation. But then John should give her 
observation greater weight as well. If they can agree on the relative weight to give their 
two observations, they can agree on their estimates regarding the car. Of course John and 
Mary might be unsure who has the better eyesight. But this is just another topic where 
they should want to combine their information, such as knowing who wears glasses, to 
form a common judgment. 

If John and Mary repeatedly exchange their opinions with each other, their opinions 
should eventually stop changing, at which point they should become mutually aware (i.e., 
have “common knowledge”) of their opinions (Geanakoplos and Polemarchakis 1982).^ 
They will each know their opinions, know that they know those opinions, and so on. 

We can now see how agreeing to disagree is problematic, given such mutual awareness. 
Consider the “common” set of all possible states of the world where John and Mary are 
mutually aware that John estimates the car age to be (i.e., has an “expected value” of) X, 
while Mary estimates it to be Y. John and Mary will typically each know many things, 
and so will know much more than just the fact that the real world is somewhere in this 
common set. But they do each know this fact, and so they can each consider, 
counterfactually, what their estimate would be if their information were reduced to just 
knowing this one fact. (Given the usual conception of information as sets of possible 
worlds, they would then each know only that they were somewhere in this common set of 
states.) 


^ For more on common knowledge, see Geanakoplos (1994), Bonnanno and Nehring 
(1999) and Feinberg (2000). For a critical view, see Koppl and Rosser (2002). For the 
related literature on "no-trade" theorems, see Milgrom and Stokey (1982). 
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Among the various possible states contained within the common set, the actual John may 
have very different reasons for his estimate of X. In some states he may believe that he 
had an especially clear view, while in others he may be especially confident in his 
knowledge of cars. But whatever the reason, everywhere in the common set John’s 
estimate has the same value X. Thus if a counterfactual John knew only that he was 
somewhere in this common set, this John would know that he has some good reason to 
estimate X, even if he does not know exactly what that reason is. Thus counterfactual 
John’s estimate should be X. 

Similarly, if a counterfactual Mary knew only that she was somewhere in the common 
set, her estimate should be Y. But if counterfactual John and Mary each knew only that 
the real world is somewhere in this common set of possible worlds, they would each have 
exactly the same information, and thus should each have the same estimate of the age of 
the car. If John estimates the car to be five years old, then so should Mary. This is 
Aumann's (1976) original result, that mutual awareness of opinions requires identical 
opinions.^ 

The same argument applies to any dispute about a claim, such as whether the car is a 
Ford, which is true in some possible worlds and false in others. As long as disputants can 
imagine self-consistent possible worlds in which each side is right or wrong, and agree on 
what would be true in each world, then it should not matter whether the disputed claim is 
specific or general, hard or easy to verify, or about physical objects, politics, or morality. 

A more detailed analysis says not only that people must ultimately agree, but also that the 
discussion path of their alternating expressed opinions must follow a random walk. Mary 

^ An argument for the irrationality of agreeing to disagree was also implicit in the classic 
"Dutch book" arguments for Bayesian rationality. These arguments showed that if an 
agent is willing to take bets on either side of any proposition, then to avoid guaranteed 
losses, his betting odds must satisfy the standard probability axioms. An analogous 
argument applies to a group of agents. If a group is to avoid combinations of bets that 
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should not be able to tell John how John’s next opinion will differ from what Mary just 
said. Mary’s best publie estimate of John’s next estimate must instead equal Mary’s 
eurrent best estimate (Hanson 2002). 

Yet in ordinary practice, as well as in controlled laboratory experiments (Hanson and 
Nelson 2004), we know that disagreement is persistent. That is, people can and do 
consistently and publicly predict the direction of other people’s opinion relative to their 
own opinion. For instance, if John first says the car is six years old, and Mary then says 
the car is three years old, a real Mary can usually accurately predict that John's next 
estimate will probably be more than three years. If Mary is rational, this suggests that 
John is not efficiently using the information contained in Mary's forecast. 

IV. Generalizations of the Basic Theory 

While Aumann’s results depended on many strong assumptions, similar results obtain 
when these assumptions are considerably relaxed. For example, rather than knowing the 
exact values of each other’s estimates, John and Mary need only be mutually aware of the 
fact that John thinks the car is at least as old as Mary thinks it is. (That is, a mutual 
awareness of the fact that X >= Y also implies that X=Y.) Larger groups of people need 
only identify the “extremist" among them, such as the person who has highest estimate 
(Hanson 1998). It is also enough for people to be mutually aware of a single summary 
statistic that increases whenever any person’s estimate increases (McKelvey and Page 
1986). 

We also can relax the requirement that John and Mary be absolutely sure of the things 
they are mutually aware of, i.e., that they have “common knowledge.” We need instead 
assume only “common belief.” That is, we need only assume that there is some common 
set of possible states of the world where 1) some condition like X>=Y holds, and 2) both 
John and Mary believe that they are in this common set. John and Mary can sometimes 


guarantee losses for the group as a whole, each group member must offer the same odds 
on every proposition. 
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be mistaken in this belief, but the higher their confidence, the smaller can be the 
difference between their estimates X and Y (Monderer and Samet 1989). 

Thus John and Mary need not be absolutely sure that they are both honest, that they heard 
each other correctly, or that they interpret language the same way. Furthermore, if John 
and Mary each assign only a small chance to such confounding factors being present, 
then their difference of opinion must also be proportionately small. This is because while 
payoff asymmetries can induce non-linearities in actions , the linearity of probability 
ensures linearity in beliefs . A rational Mary’s estimate of the car’s age must be a linear 
weighted average of her estimate conditional on confounding factors being present, and 
her estimate conditional on the absence of such factors. 

These results are also robust to John and Mary having many internal biases and 
irrationalities, as long as they also have a "rational core." Consider a corporation with 
many irrational employees, but with a truth-seeking Bayesian CEO in charge of its 
official statements. This CEO should treat inputs from subordinates as mere data, and 
try to correct for their biases. While such corrections would often be in error, this 
company’s official statements would be rational, and hence would not agree to disagree 
with statements by other companies with similar CEOs. Similarly, if John and Mary 
were mutually aware of having "clear head" rational cores capable of suspecting bias in 
inputs from the rest of their minds, they should not disagree about the car. 

We also need not assume that John and Mary know all logical truths. Through the use of 
“impossible possible states,” Bayesians do not need to be logical omniscient (Hintikka 
1975; Garber 1983). John and Mary (or their rational cores) do not even need to be 
perfect Bayesians, as similar results have been proven for various less-than-Bayesian 
agents (Rubinstein and Wolinsky 1990, Samet 1990, Geanakoplos 1994). Eor example, 
agents whose beliefs are represented by sets of probability distributions can be said to 
agree to disagree when they are mutually aware that their sets do not overlap (Eevi 1974). 
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The beliefs of real people usually depend not only on their information about the problem 
at hand, but also on their mental context, such as their style of analysis, chosen 
assumptions, and recent thoughts. The existence of such features, however, is not by 
itself a reason to disagree. A truth-seeker who does not know which mental context is the 
most reliable should prefer to average over the estimates produced in many different 
mental contexts, instead of relying on just one random context.^ So John should pay 
attention to Mary's opinion not only because it may embody information that John does 
not have, but also because it is the product of a different mental context, and John should 
want to average over as many mental contexts as he can. 

This intuition can be formalized. Assume Mary has limited computational powers. 
Regardless of the computational strategies she uses, we can call Mary a "Bayesian 
wannabe" if she can imagine counterfactually being a Bayesian, and if she wants her 
actual estimates to be as close as possible to the estimates she would have if she were a 
Bayesian. It turns out that Bayesian wannabes who make a few simple calculations, and 
who would not agree to disagree about a state-independent variable, cannot agree to 
disagree about any matter of fact (Hanson 2003). Private information is irrelevant to 
estimating state-independent variables, since they taken on exactly the same value in 
every possible state. 

V. Comparing Theory and Phenomena 

The stylized facts of human disagreement are in conflict with the above theory of 
disagreement. People disagree, yet this theory says they should not. How can we resolve 
this conflict? 

The theory above implicitly assumed that people say what they believe. Do people 
instead usually lie and not honestly state their opinions? Unfortunately for this 


’ John may have information suggesting that his mental context is better than random, but 
Mary may also have information on this topic. Persistent disagreement on this meta-topic 
should be no less problematic. 
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hypothesis, people usually have the strong impression that they are not lying, and it hard 
to see how people could be so mistaken about this. While there is certainly some element 
of sport in debates, and some recognition that people often exaggerate their views for 
effect, most people feel that they believe most of what they say when they disagree. 
People sometimes accuse their opponents of insincerity, but rarely accept this same label 
as a self-description. Even when they are conscious of steering a conversation away from 
contrary evidence, people typically perceive that they honestly believe the claims they 
make. 

Another possibility is that most people simply do not understand the theory of 
disagreement. The arguments summarized above are complex in various ways, after all, 
and recently elaborated. If this is the problem, then just spreading the word about the 
theory of disagreement should eliminate most disagreement. One would then predict a 
radical change in the character of human discourse in the coming decades. The reactions 
so far of people who have learned about the theory of disagreement, however, do not lend 
much support to this scenario. Not only do such people continue to disagree frequently, 
it seems hard to find any pair of them who, if put in contact, could not frequently identify 
many persistent disagreements on matters of fact. 

While Aumann’s result is robust to generalizing many of his assumptions, it is not robust 
to generalizing the assumption of common priors. Bayesians can easily disagree due to 
differing priors, regardless of whether or not they have differing information, mental 
contexts, or anything else. Does this allow typical human disagreement to be rational? 

To answer this question, we would need to not only identify the prior differences that 
account for typical human disagreements, we would also have to decide if these prior 
differences are rational. And this last topic turns out to be very controversial. We will 
soon review some arguments on this topic, but we will review them in the service of a 
more modest goal: evaluating whether typical human disagreement is honest. To evaluate 
the honesty of disagreement, we do not need to know what sorts of differing priors are 
actually rational, but only what sorts of differences people think are rational. We will 
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call disagreements dishonest when they are primarily the result of disputants who 
systematically violate the rationality standards that they profess and hold up for others. 

VI. Proposed Rationality Constraints On Priors 

Before reviewing arguments on the rationality of differing priors, let us review the nature 
of a Bayesian prior and prior-based disagreement. In general, Bayesian agents can have 
beliefs not only about the world, but also about the beliefs of other agents, about other 
agent’s beliefs about other agents, and so on. When modeling agents for some particular 
purpose, the usual practice is to collect a “universe” of all possible states of the world that 
any agent in the model considers, or suspects that another agent may consider, and so on. 

It turns out that one can always translate such agent beliefs into a “prior” probability 
distribution for each agent (Aumann 1998, Gul 1998). An agent's prior describes the 
probability she would assign to each possible state if her information were reduced to 
knowing only that she was somewhere in that model’s universe of states. Many dynamic 
models contain an early point in time before the agents acquire their differing private 
information. In such models, the prior is intended to describe each agent’s actual beliefs 
at this earlier time. In models without such an early time, however, the prior is 
interpreted counterfactually, as describing what agents would believe if sufficiently 
ignorant.^ 

By the nature of a prior, no agent can be uncertain about any other agent’s prior; priors 
are by definition common knowledge among Bayesians. Thus when priors differ, all 
agents know those differences, know that they all know them, and so on. So while 
agents with differing priors can agree to disagree, they must anticipate such 
disagreements. Not only are they mutually aware that they disagree, they are also 
mutually aware that their disagreement is not due to the private information they each 

o 

Note that even when priors are interpreted counterfactually, they have as much standing 
to be considered “real” as any other construct used to explain or justify spoken human 
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hold, whether that be information on the topic or information on their reasoning abilities. 
Each Bayesian knows exactly what every other agent would estimate if they had his 
information, and knows that this difference fully explains their disagreements. Relative 
to this estimate, he cannot publicly predict the future beliefs of another agent.^ 

Differing priors can clearly explain some kinds of disagreements. But how different can 
rational priors be? One extreme position is that no differences are rational (Harsanyi 
1983, Aumann 1998). The most common argument given for this common prior position 
is that differences in beliefs should depend only on differences in information. If John 
and Mary were witnesses to a crime, or jurors deciding guilt or innocence, it would be 
disturbing if their honest rational beliefs — the best we might hope to obtain from them — 
were influenced by personal characteristics unrelated to their information about the 
crime. They should usually have no good reason to believe that the non-informational 
inputs into their beliefs have superior predictive value over the non-informational inputs 
into the beliefs of others. 

Another extreme position is that a prior is much like a utility function: an ex post 
reconstruction of what happens, rather than a real entity subject to independent scrutiny. 
According to this view, one prior is no more rational than another than one utility 
function is more rational than another.If we think we are questioning a prior, we are 
confused; what we are questioning is not a prior, but some sort of evidence. In this view 
priors, and the disagreements they produce, are by definition unquestionable.^^ 


opinions, such as information sets, or epistemic principles. All such constructs are 
intrinsically counterfactual. 

^ This follows trivially from (Hanson 2002). 

One can accept this premise and still argue that priors should be treated as common. 
Given a prior, information set, and utility function that predict an agent’s choices, one 
can predict those choices as well with any other prior, as long as one makes matching 
changes to their state-dependent utility. So one can argue that it is a convention of our 
language to describe agent differences in terms of differing utilities and information, 
rather than differing priors. 

Some (Bernheim 1986, Morris 1995) argue that multiple equilibria provide a rationale 
for differing priors, since each equilibrium describes different priors over game actions. 
In each equilibrium, however, agents would agree on the prior. 
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In the vast majority of disputed topics, the available evidence does not pin down with 
absolute certainty what we should believe. A consequence of this is that if there are no 
constraints on which priors are rational, there are almost no constraints on which beliefs 
are rational. People who think that some beliefs are irrational are thus forced to impose 
constraints on what priors are rational. 

For example, technically we can think of a person at different moments in time as 
different agents, and we can even think of the different mental modules within a person’s 
mind specializing in different mental tasks as different agents (Fodor 1983). If each 
different mental module at a different time could rationally have arbitrarily differing 
priors, then almost any sequence of belief statements a person might make might count as 
rational. Those who think that some sequences of statements are irrational must thus 
impose limits on how much priors can differ for agents close in space and time. 

For example, it is common to require that the different mental modules within a single 
person share the same prior. Since it is infeasible for mental modules to share more than 
a limited amount of information with each other, we understand that different mental 
modules will sometimes give conflicting answers due to failing to share relevant 
information. Conflicts due to differing priors, however, seem less tolerable. 

As another example, it is common to require Bayesians to change their beliefs by 
conditioning when they learn (or forget) information. That is, consider an earlier "self" 
who is the immediate causal ancestor of a later "self" who has learned a new fact about 
the world. While these different selves are logically two different agents who can in 
principle have different preferences and beliefs, it is common to say that the beliefs of the 
later self should typically be equal to the beliefs of the earlier self, conditional on that 
new fact. This is equivalent to saying that these two selves should base their beliefs on 
the same prior. 


1 9 

For more on rationality constraints for a single individual over time, see Van Fraassen 
(1984), Christensen (2000), Hurley (1989), Goldstein (1985), and Gilboa (1997). 
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Why exactly should these two selves have the same prior? If it were because one self is 
the immediate causal ancestor of the other, then by transitivity all causal ancestors and 
descendants should have the same prior. And if the process of conception, connecting 
parents and children, were a relevant immediate causal relation, then since all humans 
share a common evolutionary ancestor, all humans would have to have a common prior. 

Most people do not go this far, and think that even if rationality requires a person to 
maintain the same prior over his lifespan and across his mental modules, it is also rational 
for humans to have differing priors at conception. This view, however, runs into the 
problem that it seems hard to believe that people are endowed at conception with DNA 
encoding detailed context-specific opinions on most of the topics on which they can later 
disagree. While there does seem to be a genetic component to some general attitudes 
(Olson, Vernon, Harris, & Jang 2001), the number of topics on which most people are 
capable of having independent detailed opinions is far larger than the number of bits in 
human DNA. Thus environmental influences must make an enormous contribution to 
human beliefs. 

Even so, people do seem to be endowed early on with a few beliefs that could plausibly 
explain most of their disagreements. In particular, people tend to believe that they are 
better informed and better at resisting cognitive biases that other people. Psychologists 
explain human belief formation as due to not only to general attitudes, information, and 
experiences, but also to various random features of how exactly each person is exposed to 
a topic. People are influenced by their mood, how a subject was first framed, what other 
beliefs were easily accessible then, and so on. These random initial influences on beliefs, 
when combined with the tendency of each person to think he reasons better, can easily 
produce an unending supply of independent persistent disagreements. 

How rational are such disagreements? Imagine that John believes that he reasons better 
than Mary, independent of any evidence for such superiority, and that Mary similarly 
believes that she reasons better than John. Imagine further that John accepts self-flattery 
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as a general pattern of human behavior, and so accepts it applying to himself 
counterfactually. That is, John can imagine counterfactually that he might have been 
Mary, instead of being John, and John’s prior says that if he had been Mary, instead of 
John, he would have believed that Mary reasons better than John. Mary similarly thinks 
that if she were John, she would think that John reasons better. 

Such priors are consistent in the sense that what John thinks he would believe if he were 
Mary, is in fact what Mary believes, no matter who “is” Mary. These priors are also 
“common” in the sense that everyone agrees about what Mary will think, no matter who 
really “is” Mary. These priors are not, however, “common” in the sense required for the 
theory of disagreement. Are such differing priors rational? 

One argument against the rationality of such priors is that they violate “indexical 
independence.” A non-indexical description of the state of the world includes facts like 
John and Mary’s height and IQ. An indexical description of the world, in addition, says 
things like whether the “I” that would ordinary say, “I am John,” instead says, “I am 
Mary.” Indexical descriptions and information are need to, for example, describe what 
someone doesn’t know when they have amnesia. 

Under our ordinary concepts of physical causation, we expect to be able to predict non- 
indexical features of the world using only a rich enough set of other non-indexical 
features. For example, how likely John is to be right in his current argument with Mary 
may depend on John and Mary’s experience, IQ, and education, but given a rich enough 
set of such relevant features, we do not expect to get more predictive ability from 
indexical information about who really “is” Mary. While we can imagine certain 
hypothetical scenarios where such predictive ability might arise, such as when John has 
amnesia but still knows he is very smart, these scenarios do not seem appropriate for 
priors. 

Indexical independence is the assumption that John should behave the same no matter 
who really “is” John, and similarly for Mary or anyone else (Hanson 2004). And this 
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assumption is clearly violated by John’s prior when it says that if he is John, John reasons 
better than Mary, but that if he is Mary, then Mary reasons better than John. 

Finally, some theorists use considerations of the causal origins of priors to argue that 
certain prior differences are irrational. If John and Mary have different priors, they 
should realize that some physical process produced that difference. And if that difference 
was produced randomly or arbitrarily, it is not clear that John and Mary should retain it. 
After all, if John realized that some sort of memory error had suddenly changed a belief 
he had held for years, he would probably want to fix that error (Talbott 1990). So why 
should he be any more accepting of random processes that produced his earliest beliefs? 

These intuitions can be formalized. One can argue that a rational Mary should be able to 
form coherent, even if counterfactual, beliefs about the chance that nature would have 
assigned her a prior different from the one she was actually given. Such counterfactual 
beliefs can be described by a “pre-prior.” One can argue that Mary’s actual prior should 
be consistent with her pre-prior in the sense that her prior should be obtained from her 
pre-prior by updating on the fact that nature assigned her a particular prior. Even if John 
and Mary have different pre-priors, if Mary thinks that it was just as likely that nature 
would have switched the assignment of priors, so that John got Mary’s prior and vice 
versa, then John and Mary’s priors should be the same. The priors about some event, like 
the car being a certain age, should also be the same if John and Mary believe that the 

13 

chance of getting each prior was independent of this event (Hanson 2001). 

In summary, prior-based disagreements should be fully anticipated, and there are many 
possible positions on the question of when differing priors are rational. Some say no 
differences are rational, while others say all differences are rational. Many require agents 
close in space and time to have the same prior, but allow priors to differ at conception. 
While DNA cannot encode all future opinions, it might well encode a belief that you 
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These assumptions about the causal origins of priors are satisfied by standard models 
of genetic inheritance, which predict that siblings (and parents and children) have almost 
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reason better, which could produce endless disagreements when combined with random 
influences on beliefs. Such a prior violates indexical independence, however, the idea 
that Mary’s behavior may depend on her IQ, but not on who really “is” Mary. Rational 
prior differences are also limited when your prior must be consistent with your beliefs 
about the causal origins of your prior. 

VII. Commonly Upheld Rationality Standards 

Most people have not directly declared a position on the subject of what kinds of prior 
differences are rational. Most people would have to exert great effort to even understand 
these positions. So how can we figure out which rationality positions people uphold? 

People are often reluctant to criticize the opinions of others, such as their boss to his face. 
There are, however, many situations in which people feel much freer to criticize. And in 
these situations, people often complain not about specific opinions, but about 
unreasonable patterns of opinions, patterns that seem to indicate faulty thinking. Thus 
one window into the rationality standards that people uphold and profess is the criticisms 
they make of others. 

For example, people who feel free to criticize consistently complain when they notice 
someone making a sequence of statements that is inconsistent or incoherent. They also 
complain when they notice that someone’s opinion does not change in response to 
relevant information. These patterns of criticism suggest that people uphold rationality 
standards that prefer logical consistency, and that prefer common priors for the mental 
modules within a person, and for his selves at nearby times. 

Perhaps even more frequently, people criticize others when their opinions appear to have 
self-serving biases. For example, consider those sociologists, half of who expect to 
become among the top ten leaders in their field. Consider a school administrator who 


the same ex ante chance of getting any particular DNA, and that these chances are 
correlated with little else. 


20 





favors his son for a school award, or a judge who does not excuse himself from a case in 
which he has an interest. Consider a manager who assigns himself to make an important 
sales presentation, or who uses his own judgment in an important engineering decision, 
rather than relying on apparently more qualified subordinates. 

In such cases, interested observers who feel free to criticize commonly complain about 
self-favoring beliefs. The complaint is usually not that people say things they do not 
believe, but rather that they honestly believe things that favor them, without having 
sufficient reasons for such beliefs. Though critics acknowledge that self-favoring belief 
is a natural tendency, such critics do not seem to endorse those beliefs as accurate or 
reliable. Critics warn others, for example, not to be overly influenced to share such 
biased beliefs. 

These common criticisms suggest that most people implicitly uphold rationality standards 
that disapprove of self-favoring priors, such as priors that violate indexical independence. 
These criticisms also suggest that people in fact tend to form beliefs as if they had such 
priors. That is, people do seem to think they can reason substantially better than others, 
in the absence of much evidence favoring this conclusion. People thus seem to violate 
the rationality standards they uphold. And as we have mentioned, such tendencies seem 
capable of explaining a great deal of human disagreement. 

VIII. Truth-Seeking and Self-Deception 

If we typically accept a rationality standard that disapproves of self-favoring priors, then 
why do we violate this standard with such enthusiasm? While people are usually 
embarrassed to learn they have been logical inconsistent, and smarter people do this less 
often, disagreements rarely embarrass us, and smarter people disagree just as often as 
others.^"* 


It is perhaps unsurprising that most people do not always spend the effort required to 
completely overcome known biases. What may be more surprising is that people do not 
simply stop disagreeing, as this would seem to take relatively little effort. 
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Non-truth-seeking and self-deception offer two complementary explanations for this 
difference in behavior. First, believing in yourself can be more functional that believing 
in logical contradictions. Second, while it is hard to deny that you have stated a logical 
contradiction, once the contradiction is pointed out, it is much easier to deny that a 
disagreement is due to your having a self-favoring prior. (You can always blame the 
other guy.) 

On truth-seeking, while unbiased beliefs may be closer to the truth, self-favoring beliefs 
can better serve other goals. The virtues of self-confidence and self-esteem are widely 
touted (Benabou and Tirole 2002). Parents who believe in their children care more for 
them, and the best salesmen believe in their product, whether it is good or bad. By 
thinking highly of himself, John may induce Mary to think more highly of John, making 
Mary more willing to associate with John. 

Scientists with unreasonably optimistic beliefs about their research projects may work 
harder and thus better advance scientific knowledge (Everett 2001; Kitcher 1990). 

Instead of simply agreeing with some standard position, people can better show off their 
independence and intelligence by inventing original positions and defending them. In 
response to our informal queries, numerous academics have told us that trying to disagree 
less would feel dishonest, destroy their identity, make them less human, and risk 
paralyzing self-doubt. 

Self-favoring priors can thus be “rational” in the sense of helping one to achieve familiar 
goals, even if they are not “rational” in the sense of helping one to achieve the best 
possible estimate of the true situation (Caplan 2000). 

Regarding self-deception, people seem more likely to gain the benefits of biased beliefs if 
they do not believe that they are biased (Taylor 1989). For example, a salesman is more 
persuasive when thinks he likes his product because of its features, rather that the fact 
that it is his product. And people do seem to often be unaware that they think highly of 
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themselves because of their prior. If Mary asks John to explain his high opinion of 
himself, John will usually point to some objective evidence, such a project he did well on. 
In response, John’s critics will complain that he has succumbed to wishful thinking and 
self-deception. 

Even if John attempts at some conscious levels to be unbiased, at other levels his mental 
programs may systematically bias his beliefs in the service of other goals. Our mental 
programs may under-emphasize evidence that goes against favored ideas, and distract the 
critical mechanisms that make us so adept at noticing and complaining about biases in 
other people’s opinions (Mele 2001). The great and widely recognized power of flattery 
clearly shows that people have an enormous and widely recognized capacity for self- 
deception. 

Which of these two explanations, non-truth-seeking and self-deception, is more 
fundamental? One important clue comes from the fact that academics who accept the 
conclusion that disagreement is irrational still disagree, including among themselves. 
When forced to overcome their self-deception and confront the issue, people consistently 
choose to continue to disagree. This suggests that, while this is not a conclusion they 
prefer to dwell on, most people fundamentally accept not being a truth-seeker. 

The story we have outlined so far, of a widely recognized tendency toward self-favoring 
beliefs in others, together with self-deception about this tendency in ourselves, is 
commonly told in psychology and philosophy. Evolutionary arguments have even been 
offered for why we might have evolved to be biased and self-deceived.^^ 


Many have considered the evolutionary origins of self-deception and excess confidence 
one’s own abilities (Waldman 1994). Eor example, truth-seekers who find it hard to lie 
can benefit by changing their beliefs (Trivers 1985; Trivers 2000). On topics like politics 
or religion, which are widely discussed but which impose few direct penalties for 
mistaken beliefs, our distant ancestors may have mainly demonstrated their cleverness 
and knowledge by inventing original positions and defending them well (Miller 2000). 
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This story is also commonly told in literature. For example, the concluding dream in 
Fyodor Dostoevsky's (1994 [1866]) Crime and Punishment seems to describe 
disagreement as the original sin, from which arises all other sins. In contrast, the 
description of the Houyhnhnms in Jonathan Swift’s (1962 [1726]) Gulliver’s Travels can 
be considered a critique showing how creatures (intelligent horses in this case) that agree 
too much lose their “humanity.” 

Given this story’s ubiquity, its innate plausibility, and the fact that it fits the stylized facts 
of disagreement reasonably well, let us now accept this story as a working hypothesis to 
explain most human disagreement, and consider its implications. 

VIII. How Few Meta-Rationals? 

We can call someone a truth-seeker if, given his information and level of effort on a 
topic, he chooses his beliefs to be as close as possible to the truth. A non-truth seeker 
will, in contrast, also put substantial weight on other goals when choosing his beliefs. Let 
us also call someone meta-rational if he is an honest truth-seeker who chooses his 
opinions as if he understands the basic theory of disagreement, and abides by the 
rationality standards that most people uphold, which seem to preclude self-favoring 
priors. 

The theory of disagreement says that meta-rational people will not knowingly have self- 
favoring disagreements among themselves. They might have some honest disagreements, 
such as on values or on topics of fact where their DNA encodes relevant non-self- 
favoring attitudes. But they will not have dishonest disagreements, i.e., disagreements 
directly on their relative ability, or disagreements on other random topics caused by their 
faith in their own superior knowledge or reasoning ability. 

Our working hypothesis for explaining the ubiquity of persistent disagreement is that 
people are not usually meta-rational. While several factors contribute to this situation, a 
sufficient cause that usually remains when other causes are removed is that people do not 
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typically seek only truth in their beliefs, not even in a persistent rational core. People 
tend to be hypocritical in have self-favoring priors, such as priors that violate indexical 
independence, even though they criticize others for such priors. And they are reluctant to 
admit this, either publicly or to themselves. 

How many meta-rational people can there be? Even if the evidence is not consistent with 
most people being meta-rational, it seems consistent with there being exactly one meta- 
rational person. After all, in this case there never appears a pair of meta-rationals to 
agree with each other. So how many more meta-rationals are possible? 

If meta-rational people were common, and able to distinguish one another, then we 
should see many pairs of people who have almost no dishonest disagreements with each 
other. In reality, however, it seems very hard to find any pair of people who, if put in 
contact, could not identify many persistent disagreements. While this is an admittedly 
difficult empirical determination to make, it suggests that there are either extremely few 
meta-rational people, or that they have virtually no way to distinguish each other. 

Yet it seems that meta-rational people should be discernable via their conversation 
style. We know that, on a topic where self-favoring opinions would be relevant, the 
sequence of alternating opinions between a pair of people who are mutually aware of 
both being meta-rational must follow a random walk. And we know that the opinion 
sequence between typical non-meta-rational humans is nothing of the sort. If, when 
responding to the opinions of someone else of uncertain type, a meta-rational person acts 
differently from an ordinary non-meta-rational person, then two meta-rational people 
should be able to discern one another via a long enough conversation. And once they 
discern one another, two meta-rational people should no longer have dishonest 
disagreements. 


Aaronson (2004) has shown that regardless of the topic or their initial opinions, any 
two Bayesians have less than a 10% chance of disagreeing by more than a 10% after 
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Since most people have extensive conversations with hundreds of people, many of whom 
they know very well, it seems that the fraction of people who are meta-rational must be 
very small. For example, given people, a fraction/of whom are meta-rational, let each 
person participate in C conversations with random others that last long enough for two 
meta-rational people to discern each other. If so, there should be on average/ CN/2 pairs 
who no longer disagree. 

If, across the world, two billion people, one in ten thousand of who are meta-rational, 
have one hundred long conversations each, then we should see one thousand pairs of 
people with only honest disagreements. If, within academia, two million people, one in 
ten thousand of who are meta-rational, have one thousand long conversations each, we 
should see ten agreeing pairs of academics. And if meta-rational people had any other 
clues to discern each another, and preferred to talk with one another, there should be far 
more such pairs. Yet, with the possible exception of some cult-like or fan-like 
relationships, where there is an obvious alternative explanation for their agreement, we 
know of no such pairs of people who no longer disagree on topics where self-favoring 
opinions are relevant 

We therefore conclude that unless meta-rationals simply cannot distinguish each other, 
only a tiny non-descript percentage of the population, or of academics, can be meta- 
rational. Either few people have truth-seeking rational cores, and those that do cannot be 
readily distinguished, or most people have such cores but they are in control infrequently 
and unpredictably. Worse, since it seems unlikely that the only signals of meta¬ 
rationality would be purely private signals, we each seem to have little grounds for 
confidence in our own meta-rationality, however much we would like to believe 
otherwise. 

IX. Personal Policy Implications 


exchanging about a thousand bits, and less than a 1% chance of disagreeing by more than 
a 1% after exchanging about a million bits. 
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Readers need not be concerned about the above conclusion if they have not accepted our 
empirical arguments, or if they are willing to embrace the rationality of self-favoring 
priors, and to forgo criticizing the beliefs of others caused by such priors. Let us assume, 
however, that you, the reader, are trying to be one of those rare meta-rational souls in the 
world, if indeed there are any. How guilty should you feel when you disagree on topics 
where self-favoring opinions are relevant? 

If you and the people you disagree with completely ignored each other’s opinions, then 
you might tend to be right more if you had greater intelligence and information. And if 
you were sure that you were meta-rational, the fact that most people were not might 
embolden you to disagree with them. But for a truth-seeker, the key question must be 
how sure you can be that you, at the moment, are substantially more likely to have a 
truth-seeking, in-control, rational core than the people you now disagree with. This is 
because if either of you have some substantial degree of meta-rationality, then your 
relative intelligence and information are largely irrelevant except as they may indicate 
which of you is more likely to be self-deceived about being meta-rational. 

One approach would be to try to never assume that you are more meta-rational than 
anyone else. But this cannot mean that you should agree with everyone, because you 
simply cannot do so when other people disagree among themselves. Alternatively, you 
could adopt a "middle" opinion. There are, however, many ways to define middle, and 
people can disagree about which middle is best (Barns 1998). Not only are there 
disagreements on many topics, but there are also disagreements on how to best correct for 
one’s limited meta-rationality. 

Ideally we would want to construct a model of the process of individual self-deception, 
consistent with available data on behavior and opinion. We could then use such a model 
to take the observed distribution of opinion, and infer where lies the weight of evidence, 
and hence the best estimate of the truth. A more limited, but perhaps more feasible. 

Ideally this model would also satisfy a reflexivity constraint: when applied to disputes 
about self-deception it should select itself as the best model of self-deception. If most 
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approach to relative meta-rationality is to seek observable signs that indicate when people 
are self-deceived about their meta-rationality on a particular topic. You might then try to 
disagree only with those who display such signs more strongly than you do. 

For example, psychologists have found numerous correlates of self-deception. Self- 
deception is harder regarding one’s overt behaviors, there is less self-deception in a 
galvanic skin response (as used in lie detector tests) than in speech, the right brain 
hemisphere tends to be more honest, evaluations of actions are less honest after those 
actions are chosen than before (Trivers 2000), self-deceivers have more self-esteem and 
less psychopathology, especially less depression (Paulhus 1986), and older children are 
better than younger ones at hiding their self-deception from others (Feldman & Custrini 
1988). Each correlate implies a corresponding sign of self-deception. 

Other commonly suggested signs of self-deception include idiocy, self-interest, emotional 
arousal, informality of analysis, an inability to articulate supporting arguments, an 
unwillingness to consider contrary arguments, and ignorance of standard mental biases. 

If verified by further research, each of these signs would offer clues for identifying other 
people as self-deceivers. 

Of course, this is easier said than done. It is easy to see how self-deceiving people, 
seeking to justify their disagreements, might try to favor themselves over their opponents 
by emphasizing different signs of self-deception in different situations. So looking for 
signs of self-deception need not be an easier approach than trying to overcome 
disagreement directly by further discussion on the topic of the disagreement. 

We therefore end on a cautionary note. While we have identified some considerations to 
keep in mind, were one trying to be one of those rare meta-rational souls, we have no 
general recipe for how to proceed. Perhaps recognizing the difficulty of this problem can 
at least make us a bit more wary of our own judgments when we disagree. 


people reject the claim that most people are self-deceived about their meta-rationality, 
this approach becomes more difficult, though perhaps not impossible. 
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X. Conclusion 


A literature started by Robert Aumann, and spanning several deeades, has explored the 
finding that, on matters of fact, honest disagreement is problematie. We have reviewed 
this literature, and found Aumann’s initial result to be robust to many permutations, 
though not to introducing rationally differing priors. We reviewed arguments about 
which prior differences are rational, and found that the eontroversy surrounding this topic 
makes it diffieult to determine whether typieal disagreements are rational. 

We ean, however, use the rationality standards that people seem to uphold to find out 
whether typieal disagreements are honest, i.e., are in aeeord with the rationality standards 
people uphold. We have suggested that when eritieizing the opinions of others, people 
seem to eonsistently disapprove of self-favoring priors, sueh as priors that violate 
indexieal independence. Yet people also seem to consistently use sueh priors, though 
they are not inelined to admit this to themselves or others. 

We have therefore hypothesized that most disagreement is due to most people not being 
meta-rational, i.e., honest truth-seekers who understand disagreement theory and abide by 
the rationality standards that most people uphold. We have suggested that this is at root 
due to people fundamentally not being truth-seeking. This in turn suggests that most 
disagreement is dishonest. 

We presented crude ealculations suggesting that very few people ean have mueh ground 
for thinking themselves meta-rational. This faet need not eause much concern for those 
willing to embraee the rationality of self-favoring priors, and to forgo eritieizing the 
beliefs of others resulting from sueh priors. It also need not eoneem those who rejeet our 
empirieal arguments. Those who aeeept our empirieal arguments and who aspire to 
meta-rationality, however, faee more diffieulties, whieh we have briefly outlined. 
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